Empirical research methods department: Mission, learnings, and future plans

A recent trend in management journals is to involve specialized methods editors check for the quality of an article's empirical execution (Antonakis et al., 2019; Bergh & Oswald, 2020; Hardwicke 2019). The Journal Operations Management (JOM) has been engaging specialists explicitly review process about 5 years now. initial transition into a matrix organization (Guide Ketokivi, 2015), which each article assigned subject area editor (vertical) and, if necessary, also review-team member specializing method or theory (horizontal) was superseded by creation Empirical Research Methods (now shortened just Methods, ERM) department current editors-in-chief (Browning de Treville, 2018). In this editorial, we give overview operations ERM department, present collection common issues have gathered, and discuss strategic direction department. serves our research community two ways: (1) publishing manuscripts use OM (2) performing checks incoming before they go topical Both roles align with journal's evolution terms methodological rigor. We will address these turn. JOM tradition articles, including those qualitative such as case studies (Barratt 2011; Handfield Melnyk, 1998; Ketokivi Choi, 2014; Meredith, Stuart 2002), plans maintain going forward. To help carry forward encourage it even more, would like add some clarity very fundamental questions interested authors: How do determine study methodologically focused hence suitable handle? addition, article, how whether makes contribution? Recognizing that difficult define catchall standard, offer develop-review-import classification, broadly categorized three classes shown Table 1. Endogeneity (Lu 2018) Mediation (Malhotra 2014) imported already known researchers. represents too much niche research. Inattentive survey responding (Abbey Meloy, 2017) Multilevel factor analysis (Ketokivi, 2019) Articles develop category provide original contributions. Typically, solve problem techniques toolbox not fully address. Many utilizing prediction, forecasting, machine learning models part their development naturally fit under class (Chuang 2021; Ilk Ketzenberg Pak Petropoulos Whether goes depends on novelty degree contribution comes from rather than project more generally. must evaluation method's performance using tools Monte Carlo simulations and/or predictive-accuracy tests. technical aspects contributions should be written at standard required applied research-methods Organizational while maintaining contextual focus OM. well applying (e.g., Lu 2018; Malhotra 2014). typical starts comprehensive published papers focal method, then provides summary identified, follows up recommendations improvement practice. It important are supported citations evidence high-quality textbooks econometrics books) proofs other forms evidence. Authors simultaneously recall reader researcher who benefit, example, discussion exclusion condition instrumental variables sample-selection models, but greatly benefit insights regarding search identify context. al. (2018)—handled article—strikes good balance between above issues. authors did thorough studies, focusing addressed endogeneity, identified several Importantly, (2018) only explained could provided clear pointers widely used books well-known contain behind techniques. final class, import, contains articles introduce disciplines emphasis applicability technique context, demonstrated example. can either new test existing context Abbey 2017). Review import overlap: might better Prospective writing JOM, especially classes, start submitting proposal get early feedback editors. briefly describe topic explain why warrants introduction review. recently commentary experiments (Eckerd 2021) started evolved considerably moved toward publication, leading rich meaningful discussions journal This preliminary carried out prior process: Success step does guarantee publication. excited tell currently considering interesting proposals making promising progress second role support departments pre-review checks. regular typically assesses aspects: theoretical contribution, practical implications, (3) rigor design execution. first require strong knowledge relevant literature third one less so. That is, although variations exist what emphasized across disciplines, there little anything all specific Although novel constantly being introduced, many basic principles remain unchanged over time. crates opportunity standardized control checks: Manuscripts submitted regardless area, processed initially via assess (Figure 1 Browning goal any major weaknesses sent outcome is: approval move directly request revision manuscript, rejection due When requesting revision, try best clearly practically possible make publishable. Nevertheless, always risk manuscript deemed acceptable later rejected because lack contribution. For reason, done round. successfully passing prevent reveal further cases, focuses summarized past. From reviewer-development point view, method-focused reviews allow us engage early-career scholars yet experience serve reviewers JOM. requirements reviewer somewhat different reviewer. Evaluation implications requires perspective largely gained through experience. contrast, expertise certain type methodology, doctoral level course work self-learning. reviews, freshness full reports. Therefore, most ourselves, actively building pool reviewers. expand pool, extend warm invitation late-stage students junior colleagues starting academic career. interested, developed recruitment form.11 If you print copy find form here: https://tinyurl.com/jom-methods-review Doing attractive reasons. First, great practice one's own skills. no one-size-fits-all expect guidance mentoring editors, include template explaining problems, notes editor, books, materials evaluation, version review, short conference call article. Based experience, student found particularly motivating helpful developing confidence sincere hope become developmental both Second, doing boost career ways. Academic generally involves research, teaching, service. While often teaching opportunities, professional service opportunities come seniority. duties. Furthermore, way involved offers path editorial board (ERB) journal. keeps track workload performance, policy recommend completed five ERB appointment. section presents encountered during without exception quantitative studies,22 Qualitative assessed process. so purely arise issue concisely explanations. list here exhaustive, aim useful pre-submission checkpoint, hopefully saving least round revision. checklist complements replace resources American Psychological Association guidelines (Applebaum excellent Reviewer's Guide Quantitative Social Sciences (Hancock organize topics design, analysis, interpretation reporting. endogeneity classified categories, decided collect them relative frequency. intertwined, categorization merely attempt materials. rest presented issue-recommendation format. term “field experiment” sometimes loosely represent types design: randomized field experiment quasi/natural experiment. They are, however, designs drastically strategies causal identification (Shaver, 2020). analog controlled lab real world (i.e., field). key features treatment group participants groups randomly. Nguyen Kim (2019) 328 textile, clothing, leather, footwear firms control, quality-management meeting, placebo meeting conditions meetings had impact practices after 2-year lags. Chuang (2016) 60 retail stores received audits effect operational performance. natural special observational study, where units experienced others not, assignment randomized. treated quasi random, cases achieved difference-in-difference strategy, data preprocessed matching synthetic achieve comparability Arslan Other approaches regression discontinuity (Calvo (Angrist Keueger, 1991) used, permitted Use terminology precisely experiment). Chatterji (2016), Eden (2017), Lambrecht Tucker (2015) conducting experiments. Also, consult parts Eckerd (2021), Grant Wall (2009), Lonati (2018), Sieweke Santoni (2020). idea administer stimulus (typically survey) triggers turn produces (often measured same survey). variety experimental vignettes. refer readers details emphasize vignette appropriate tool studying particular question critically underlying mechanism theorization. individual motivates cognition, appear appropriate. Using vignettes organizational theory, informant take stimuli organization, wait play out, report outcome. unlikely happen given its duration. significant difference sufficient played simply implicit theories (Podsakoff 2003) demand (Lonati informants respond based think respond. Exercise extreme caution when designing outside scope processes, covered behavioral operations. Vignettes valuable answering questions. Explicitly possibility effects caused non-consequential decision-making Online Panels Amazon's Mechanical Turk (Mturk) source number Ball Cantor Jin, debate appropriateness centers around subjects possessing reliably recruited platforms (Aguinis Lee Porter elicited perception consumers, preference remanufactured products Agrawal (2015), Mturk reliable relatively low cost source. instead requested answer requiring domains marketing sales; eight total defined Mturk), abundance needs exercised ensure nontrivial fraction online panel labor indeed possesses domain expertise, qualified screened out. criterion extent subjective delicate argue. On hand, acknowledge tasks stylized fashion newsvendor model) managers perform similarly (Bolton 2012). perspective, appears relied upon conduct background possess. experiential takes inside company available level, face validity since “are unemployed underemployed mispresent qualifications” 2021). Conditional reasonable satisfaction criterion, along lines improved two-stage procedure. faced potential concern qualification (first criterion), advise proactive presenting arguments match background. process, insert prescreening administering main study. indication sought Its purpose let state few related area. Only stated “sought-for” correctly answered stage (for study). procedure screen desired truthfully lower lying otherwise. event unsuitable source, underestimate value supplemental broad accessibility low-cost. One scenario arises replication sample highly targeted population prohibitively costly rerun (Lee Another large amount manipulations (Mummolo Peterson, Quidt Last, note panels share similarities purposes, exhibit differences (Eyal 2021), implies automatically apply Qualtrics Prolific. Moving forward, samples OM, future sharpen understanding workers really are. objective demographics collected websites, social media profiles LinkedIn), calling informants. After all, warrant (Ketokivi Mantere, use. measure variable, frequently run statements “our adapted (a citation)” “we follow citation) create variable.” Citing desirable insufficient citation logical transparent operationalization captures construct uncommon cited realizes differs substantially manuscript. Not scale adaptation negative impression paper, risks contaminating untested questionable scales. Be describing adaptations scales previous items identical literature, (and item quotation marks). Otherwise, appendix table shows exact wordings (in marks) compared adaptation. Arguing cross-sectional advised: Such survive An (2015, 4.1) blanket recommendation: “For researchers, simple single-informant surveys stop claims bias.” points. skeptical post hoc unless marker employed (Richardson 2009). general “procedural remedies” anonymity eliminate problem, collecting multiple sources certainly mitigates problem. Third, variance beyond single assumed adequately effects, affected (Spector dual having success authors) sparing journal), join advising researchers surveys, unique rate past close zero contrast usual extensive explanation choice dependent independent hypotheses), tend invest variables. controls explanation, inclusion sentence like: “Following citations), …” expected receive attention variables, variable alternative correlation influence (Spector, β , X Y, roles. consider 2 correlated captured . means tied proxied (with quality), recommended regression, ensures unbiasedness ordinary squares [OLS] regression) consistency33 “Unbiasedness” “consistency” interchangeably authors, different. Loosely stated, unbiased estimator ̂ mean equal finite size = 1000). consistent estimator, asymptotically infinity). estimators nonlinear necessarily unbiased. OLS estimates linear consistent. models) estimated unfortunately cannot measured, gives rise omitted bias estimating —the reason section. Thus, discussed together concerns strategy. Specifically, absence careful thought plausible bias, effort proxies, likely referred “control approach” addressing See 4.2 Shang (2017) example implementation. illustrate role-played 3 uncorrelated Y affect Ε [ ] change included not. However, explains variation error function unexplained R models), decreases. detecting significance easier. Note contrary, irrelevant precise. Ground perform. citing needed. As rule thumb, fulfilling roles, 2010). Additional surrounding Becker Spector (2021, side note, aware bad (Cinelli influenced creates “collider bias” (Pearl Mackenzie, 2018, pp. 185–186), invalidates inference Sample selection observed selected subsample entire population, missing random (MNAR) (Enders, 2010, Chap. 1) prerequisite—perhaps obvious one—for exist, bias. additional satisfied. interest, X, predictor equation sense). equations correlate. Certo detailed discussion. coefficients correlations, (a.k.a. Missing MAR) lead biased estimates. tested coefficient (Certo 2016), condition, strictly speaking, empirically testable unobserved. indirect test—checking denoted ρ)—critically relies bivariate normal functional assumption preferably restriction equation. result, tests trades untestable (error correlated) another distribution).44 situation analogous essentially shifts exogeneity (assumed) endogenous instrument.